Apparatus and method for displaying images

ABSTRACT

A mask region extraction unit extracts a region that is to be masked in a panoramic image. A mask processing unit generates the object image where the region to be masked in the panoramic image is masked. A positioning unit adjusts the direction of a spherical image to the shooting direction of the object image. A mapping processing unit maps the mask-processed object image and the spherical image onto a three-dimensional (3D) object space as textures. A 3D image generator generates a 3D panoramic image, when the 3D panoramic image is viewed in a specified line of sight in such a manner so as to regard the shooting location of the panoramic as a viewpoint position.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an apparatus and a method fordisplaying images.

2. Description of the Related Art

With the prevalence of digital still cameras and digital video cameras,there are increased occasions where still images or moving images havingbeen shot are stored in a computer for later viewing, processing, ordisplaying on the screen of a game device or a television system. It isalso popularly done that the shot moving images are uploaded to aposting site on the Internet so as to share them with the other users.

Among the digital cameras are those capable of shooting panoramicimages, which allow the image taking of panoramic images of wide viewangle with perfect ease. Also in wide use are software tools that cangenerate a panoramic image by stitching together a plurality of imagesshot by a digital camera from different shooting directions.

There is a site named “360cities” (http://www.360cities.net) thataccepts the posting of panoramic images shot by users and show them onthe Internet, so that the users around the world can view the panoramicimages posted.

A panoramic image is an image shot as viewed in all directions from ashooting location. When such a panoramic image is shot outdoors, it isoften the case that the sky is opened up above a person taking the imageof a panoramic view and only the sky is captured on a top area of thepanoramic image shot. If such an image is processed so that some kind ofvisual effect can be created in the sky region occupying a highproportion of the panoramic image, it can be expected that the panoramicimage will be more refreshing than otherwise and a diversity will bebrought to the panoramic image. Also, something excitingly unpredictablemay be likely to happen and amusing factor may also be likely to beadded if the rendering is done such that another object is inserted intothe panoramic image shot so as to create a new panoramic imageincorporating a virtually nonexistent object rendered therein.

SUMMARY OF THE INVENTION

The present invention has been made in view of these problems, and ageneral purpose thereof is to provide a technology for processing imagesand displaying the thus processed images.

In order to resolve the above-described problems, an image displayapparatus according to one embodiment of the present invention includes:a storage configured to store an object image, associated withinformation on a shooting direction, and a spherical image, associatedwith directional information; a mask region extraction unit configuredto extract a region that is to be masked in the object image; a maskprocessing unit configured to generate the object image where the regionto be masked in the object image is masked; a positioning unitconfigured to adjust the directional information of the spherical imageto be the shooting direction of the object image; a mapping processingunit configured to position the spherical image and then configured tomap the mask-processed object image and the spherical image onto athree-dimensional (3D) object space as a texture; a three-dimensional(3D) image generator configured to generate a three-dimensional (3D)object image, when the 3D object image mapped by the mapping processingunit is viewed in a specified line of sight in such a manner so as toregard a shooting location of the object image as a viewpoint position;and a display control unit configured to display the 3D object image ona screen.

Another embodiment of the present invention relates also to an imagedisplay apparatus. The apparatus includes: a storage configured to storean object image, associated with information on a shooting direction,and a virtual image, associated with directional information; a maskregion extraction unit configured to extract a region that is to bemasked in the object image; a mask processing unit configured togenerate the object image where the region to be masked in the objectimage is masked; a positioning unit configured to adjust the directionalinformation of the virtual image to the shooting direction of the objectimage; and a display control unit configured to position the virtualimage and then configured to display on a screen an augmented-realityimage where the virtual image is superimposed onto the mask-processedimage.

Still another embodiment of the present invention relates to a methodfor displaying images. The method includes: reading out, by a processor,an object image and a spherical image to be displayed, from a storagedevice for storing the object image, associated with information on ashooting direction, and the spherical image, associated with directionalinformation; specifying, by a processor, a region that is to be maskedin the object image; adjusting, by a processor, the directionalinformation of the spherical image to the shooting direction of theobject image; and rendering, by a processor, after positioning thespherical image in a manner such that the spherical image issuperimposed onto the region to be masked in the object image.

Optional combinations of the aforementioned constituting elements, andimplementations of the invention in the form of methods, apparatuses,systems, computer programs, data structures, recording media and soforth may also be effective as additional modes of the presentinvention.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will now be described by way of examples only, withreference to the accompanying drawings which are meant to be exemplary,not limiting, and wherein like elements are numbered alike in severalFigures in which:

FIG. 1 is a configuration diagram of a panoramic image display apparatusaccording to an embodiment;

FIG. 2 shows a structure of a controller, connected to the panoramicimage display apparatus of FIG. 1, which is an example of an inputdevice;

FIGS. 3A to 3D are illustrations with which to explain the mechanism andshooting directions of an omnidirectional image shooting system used toshoot panoramic images;

FIG. 4A is an illustration with which to explain azimuth angle θ of acamera;

FIG. 4B is an illustration with which to explain elevation angle φ of acamera;

FIGS. 5A to 5C are illustrations with which to explain a panoramic imageshot when an initial position of a camera is in a direction of azimuthangle θ;

FIGS. 6A to 6C are illustrations with which to explain a panoramic imageshot when a camera is in a direction of elevation angle φ=60°;

FIG. 7A explains a method of how a panoramic image is created bystitching a plurality of images together;

FIG. 7B explains a method of how a panoramic image is created bystitching a plurality of images together;

FIG. 8 is a flowchart showing a procedure for generating a panoramicimage by the panoramic image display apparatus of FIG. 1;

FIG. 9A shows an example of a panoramic image shot;

FIG. 9B shows a panoramic image that is segmented into segmented imagesby segmentation;

FIG. 9C shows a panoramic image that has been filtered;

FIG. 9D shows a mask by which a sky region in a panoramic image isspecified;

FIG. 9E shows a panoramic image obtained after the sky region has beenmask-processed;

FIG. 10 is a diagram for explaining a relation between a sphere and a 3Dpanoramic image into which a panoramic image is mapped;

FIG. 11A shows an example where a spherical image has been pasted to thesky region of the mask-processed panoramic image of FIG. 9E; and

FIG. 11B shows an example where a starry sky image has been pasted tothe mask-processed panoramic image of FIG. 9E.

DETAILED DESCRIPTION OF THE INVENTION

A description will be given of an outline of a preferred embodiment. Aregion where the sky is captured in a panoramic image is extracted andthen a mask processing is performed on this extracted sky region to maketransparent the sky region. A sphere or celestial sphere is assumedoutside a three-dimensional (3D) panoramic space, such as a panoramicsphere, onto which a panoramic image is mapped, and a sky image and thelike are prepared beforehand as a spherical image. And the sphericalimage is mapped onto the 3D panoramic space into which the panoramicimage is mapped. This generates a 3D panoramic image where the sphericalimage has been pasted to the sky region, which is clipped and nowtransparent, in the panoramic image. Another object, which is notcaptured in the panoramic image, may be placed in the sphere and thenthe thus generated 3D panoramic image including this object may bemapped onto the 3D panoramic image, so that the object can be insertedinto the generated 3D panoramic image. Note that a plurality of spheresmay be provided and those spheres may be mapped onto the 3D panoramicspace. For example, a sphere composed of clouds and a sphere composed ofstars may be provided; images of clouds may be prepared for the sphereof clouds and celestial bodies such as the sun, the moon, and stars maybe prepared for the sphere of stars.

FIG. 1 is a configuration diagram of a panoramic image display apparatus100 according to a preferred embodiment. For example, the panoramicimage display apparatus 100 as shown in FIG. 1 may be functionallyconfigured such that hardware, software or a combination of both isimplemented to a personal computer, a game device, a portable device, amobile terminal and so forth. Part of such functional components may beimplemented in a client, so that the panoramic image display apparatus100 may be realized as a server-client system via a network.

A panoramic image/additional data storage 24 stores panoramic imageshaving information on shooting locations and information on shootingorientations associated with each other. The additional data, such asinformation on shooting locations and shooting orientations, may beadded directly to a data file of panoramic images, or may be managed asa separate file from the panoramic images.

The information on shooting locations includes, for instance,information on latitudes and longitudes which is given by GPS (GlobalPositioning System). The information on shooting orientation includes,for instance, information on the azimuth (angle of orientation) of thecenter point of a panoramic image obtained from an azimuth sensor, andmay also additionally include information on the elevation angle androll angle of a camera at the time of shooting.

If the azimuth of the center point of a panoramic image is given asinformation on a shooting orientation, then it is possible to calculatethe orientation of an arbitrary point of the panoramic image based onthe angle of panning the camera to the right or left. The panoramicimages may have, as the information on the shooting orientations, thecoordinate values of pixels in the orientations of true north, truesouth, true east, and true west of the panoramic images which arecalculated based on the azimuths and pan angles of the center points ofthe panoramic images.

A panoramic image acquiring unit 10 acquires a panoramic image to bedisplayed from the panoramic image/additional data storage 24. Thepanoramic image to be displayed is identified as the user specifies ashooting location on a map or the like.

A mask region extraction unit 11 extracts a region that is to be maskedin an object to be displayed. The region to be masked is a sky regioncaptured in a panoramic image, for instance. The sky region may bedetermined to be as such, based on color information of the panoramicimage. Since the sky is often opened up way above a person who takes theimage of a panoramic view, the sky is often detected as a region widelyextending continuously at a wide elevation angle of a camera.

Information on the elevation angle of the camera at the time of shootingis appended to the panoramic image. Thus, the mask region extractionunit 11 may determine the horizon position in the panoramic image basedon the information on the elevation angle of the camera and may extractthe sky region in an area above the horizon. Or if the elevation angleof the camera is greater than a predetermined threshold value, the areawill definitely be above the horizon and therefore the sky region may beextracted within such the area. This can avoid the error of mistakenlyextracting an area other than the area above the horizon, as the skyregion, even though there is a region, below the horizon, whose color issimilar to the color of the sky.

A mask processing unit 12 generates a panoramic image where a region, tobe masked, which is extracted by the mask region extraction unit 11 hasbeen masked. If the mask region extraction unit 11 extracts the skyregion as the region to be masked, the panoramic image will be subjectedto the mask processing with the extracted sky region as a mask so as togenerate a panoramic image with the sky region masked. The maskprocessing performed on the panoramic image is achieved such that aregion specified as the mask in the original panoramic image is madetransparent or an image with said region clipped is generated. The maskprocessing used herein may use a two-level mask, where the pixel valuesin a mask region. However, this should not be considered as limitingand, for example, a multi-level mask may be used where the pixel valuesin the mask region are alpha-blended. The mask processing unit 12 storesthe panoramic image after the mask processing in the panoramicimage/additional data storage 24.

Stored in a spherical image data storage 26 is spherical image data thatis to be superimposed onto a panoramic image after the mask processing.The sphere (celestial sphere) lies outside the 3D panoramic space intowhich the panoramic image is mapped, and an image of the sky set in thesurface of the sphere serves as a spherical image. Spherical images asviewed from various points are stored, in the spherical image datastorage 26, in association with the latitude and longitude of theviewpoints. The spherical image may contain images of celestial bodiessuch as the sun, the moon, and stars and an image of clouds, forinstance, and may also contain the moving images of celestial bodies andclouds based on a motion model of celestial body and a meteorologicalmodel, respectively. The spherical images may be those produced based onan actual picture or moving images or those produced based on computergraphics.

A positioning unit 20 adjusts the direction of a spherical image to theshooting direction of the panoramic image after the mask processing.More specifically, the center of a sphere, whose center is the latitudeand longitude, is aligned to the center of the 3D panoramic space intowhich the panoramic image is mapped. Then, for example, with thedirection of the North Star used as a reference, the rotational angle ofthe spherical image is computed so that the rotational angle thereof isequivalent to or matches the shooting direction determined by the panangle and tilt angle of the panoramic image, and then the sphericalimage is rotated by the thus computed rotational angle.

Stored in an object data storage 28 is 3D data of objects to be insertedinto a panoramic image. An object may be a static object that standsstill at a predetermined latitude, longitude and altitude and may alsobe a dynamic object that moves in a 3D space based on a motion model.

An object insertion unit 22 places an object, which is to be inserted tothe panoramic image, in a designated spatial position. The object may beplaced between the sphere and the 3D panoramic space into which thepanoramic image is mapped, namely outside the 3D panoramic space. Also,the object may be placed within the 3D panoramic space. Note that theinsertion of an object by the object insertion unit 22 is optional.

A positioning unit 20 positions the panoramic image and the sphericalimage before a mapping processing unit 14 maps the mask-processedpanoramic image and the spherical image into the 3D panoramic space astextures.

In the case of a spherical, or omnidirectional (celestial), panoramicimage, a sphere is assumed as the 3D panoramic space and a panoramicimage is texture-mapped onto the sphere by a sphere mapping. Or a cubemay be assumed as the 3D panoramic space and then a panoramic image maybe texture-mapped onto the surface of cube by a cube mapping. Also, inthe case where the panoramic image does not have any component in tiltdirections and spreads only in the panning directions, a cylinder may beassumed as the 3D panoramic space, and the panoramic image may betexture-mapped onto the surface of the cylinder by a texture mapping.The same applies to the case where the panoramic image does not have anycomponent in the panning directions and spreads only in tilt directions.

A mapping processing unit 14 maps the spherical image into the 3Dpanoramic space and then maps the panoramic image by superimposingmask-processed panoramic image onto the 3D panoramic space. Thisproduces an image, where part of the spherical image has been pasted toa mask region within the panoramic image, onto the 3D panoramic space.

If an object is placed inside or outside the 3D panoramic space by theobject insertion unit 22, the mapping processing unit 14 will also mapthe object placed thereby into the 3D panoramic space.

If the object is placed outside the 3D panoramic space, the mappingprocessing unit 14 will map the object into the 3D panoramic space andthen map the panoramic image by superimposing mask-processed panoramicimage onto the 3D panoramic space. Thereby, the object is pasted to thesky region of the panoramic image and, if the object overlaps with abuilding or the like captured in the panoramic image, the object isrendered so that the object is hidden behind or obscured by thebuilding.

If the object is placed inside the 3D panoramic space, the mappingprocessing unit 14 will map the object into the 3D panoramic space andthen map the object by superimposing object onto the 3D panoramic space.Thereby, the object is superimposed onto the 3D panoramic image and, ifthe object overlaps with a building or the like captured in thepanoramic image, the object is rendered in front of the building.

A 3D image generator 16 generates a three-dimensional (3D) panoramicimage when the 3D panoramic space having a panoramic image, a sphericalimage and an object mapped thereon by the mapping processing unit 14 isviewed in a specified line of sight. When the 3D panoramic space is asphere, the viewpoint is placed at the center of the sphere. When the 3Dpanoramic space is a cube, the viewpoint is placed at the center of theinterior of the cube. And when the 3D panoramic space is a cylinder, theviewpoint is placed on the center axis of the cylinder. The viewpoint isthe location where the panoramic image to be displayed is shot, and theline of sight is the direction in which the surrounding area is viewedand is thus identified by the azimuth and the elevation angle. The 3Dimage generator 16 generates a 3D image when the 3D panoramic space isviewed in the line of sight identified by the azimuth and the elevationangle.

A display control unit 18 has a 3D panoramic image or a map image thusgenerated displayed on a screen of the display unit.

A user interface 40 is a graphical user interface through which the usercan manipulate the graphics displayed on the screen of a display usingan input device. The user interface 40 receives user instructions on themap or 3D panoramic image displayed on the screen from the input devicewhich may be a controller of a game device, a mouse, a keyboard, or thelike. FIG. 2 shows a controller 102 as an example of the input device,whose construction will be discussed in detail later.

The user interface 40 instructs the panoramic image acquiring unit 10 toacquire a specified panoramic image from the panoramic image/additionaldata storage 24.

The user can input instructions to change the line of sight for viewingthe 3D panoramic space by operating an analog stick 118 or directionkeys 116 of the controller 102, for instance. A line-of-sight settingunit 32 of the user interface 40 gives a line of sight instructed by theuser to the 3D image generator 16. The 3D image generator 16 generatesan image when the 3D panoramic space is viewed in a specified line ofsight.

An angle-of-view setting unit 31 sets an angle of view when the user hasperformed a zoom operation on the panoramic image being displayed andgives the information of the angle of view thus set to the panoramicimage acquiring unit 10 and the 3D image generator 16. Where panoramicimages of different angles of view are stored in the panoramicimage/additional data storage 24, the panoramic image acquiring unit 10reads out a panoramic image of an angle of view closest to the set angleof view and changes the panoramic image to be displayed. The 3D imagegenerator 16 realizes the visual effects of zoom-in and zoom-out byenlarging or reducing the 3D panoramic image according to the set angleof view.

A panoramic image may have information on the shooting altitude, and thepanoramic image/additional data storage 24 may store panoramic imagesshot at different altitudes at the same shooting location. In such acase, the user can input instructions to change the altitude byoperating L1/L2 buttons 161 and 162 provided on the left front of thecasing of the controller 102, for instance. Pressing the L1 button 161will give an instruction to raise the altitude, and pressing the L2button 162 will give an instruction to lower the altitude.

The display control unit 18 may indicate to the user, for instance, withsmall arrows at the top and bottom portions of the screen that thepanoramic image currently being displayed has panoramic images shot atdifferent altitudes at the same shooting location. An arrow facingupward in the top portion of the screen indicates the presence of apanoramic image shot at a higher altitude than the current one, and anarrow facing downward in the bottom portion of the screen indicates thepresence of a panoramic image shot at a lower altitude than the currentone.

Upon receipt of an instruction from the user to change the altitude, thealtitude setting unit 34 of the user interface 40 instructs thepanoramic image acquiring unit 10 to acquire a panoramic imagecorresponding to the specified altitude, despite the same latitude andlongitude, from the panoramic image/additional data storage 24. Thepanoramic image acquiring unit 10 acquires a panoramic image of a highershooting altitude than the panoramic image currently being displayedwhen the L1 button 161 is pressed, and acquires a panoramic image of alower shooting altitude than the current one when the L2 button 162 ispressed.

When a display is produced by switching to a panoramic image of adifferent shooting altitude, the display control unit 18 may give aspecial effect to the image so that the user may have a sense of ridingan elevator up or down. For example, when switching to a panoramic imageof a higher altitude, the panoramic image currently being displayed canbe scrolled downward, thereby having the panoramic image of a higheraltitude descend from above with the result that the user may have asense of having risen upstairs.

A panoramic image contains information on the shooting date and time,and the panoramic image/additional data storage 24 may store panoramicimages shot at different dates and times at the same shooting location.In such a case, the user can input instructions to change the date andtime by operating R1/R2 buttons 151 and 152 provided on the right frontof the casing of the controller 102, for instance. Pressing the R1button 151 will give an instruction to shift to a later date and time,and pressing the R2 button 152 will give an instruction to shift to anearlier date and time.

The display control unit 18 may indicate to the user, for instance, withwatch and calendar icons in the corner of the screen that the panoramicimage currently being displayed has panoramic images shot at differentdates and times. Watch icons may be displayed to indicate the presenceof panoramic images for different times of day such as morning, noon,and night, whereas calendar icons may be displayed to indicate thepresence of panoramic images for different seasons such as spring,summer, autumn, and winter.

Upon receipt of an instruction from the user to change the date andtime, the date/time setting unit 36 of the user interface 40 instructsthe panoramic image acquiring unit to acquire a panoramic imagecorresponding to a specified date and time at the same shooting locationfrom the panoramic image/additional data storage 24. The panoramic imageacquiring unit 10 acquires a panoramic image of a later shooting dateand time than the panoramic image currently being displayed when the R1button 151 is pressed, and acquires a panoramic image of an earliershooting date and time than the current one when the R2 button 152 ispressed.

Thus, it is possible to switch the panoramic image being displayed topanoramic images of a different time of day or season at the sameshooting location, for example, from one shot in the morning to one shotat night, or from one shot in spring to one shot in autumn. In changingthe panoramic image, the display control unit 18 may give an effect offade-in and fade-out to the image.

A viewpoint position setting unit 30 set the shooting location of apanoramic image as a viewpoint position and conveys it to the 3D imagegenerator 16. The line-of-sight setting unit 32 sends the specifiedline-of-sight to the 3D image generator 16.

FIG. 2 shows a structure of a controller, connected to the panoramicimage display apparatus of FIG. 1, which is an example of an inputdevice. The panoramic image display apparatus 100 may be a game device,for instance.

The controller 102 has a plurality of buttons and keys to receivecontrol inputs to the panoramic image display apparatus 100. As the useroperates on the buttons or keys of the controller 102, their operationinputs are transmitted to the panoramic image display apparatus 10through wireless or wired connections.

Provided on a casing top surface 122 of the controller 102 are a groupof arrow keys 116, analog sticks 118, and a group of operation buttons120. The group of direction keys 116 include “up-”, “down-”, “left-”,and “right-” direction indication keys. The group of operation buttons120 include a circle button 124, a cross button 126, a square button128, and a triangle button 130.

The user holds a left-hand grip 134 b with the left hand and holds aright-hand grip 134 a with the right hand, and operates the group ofdirection keys 116, the analog sticks 118, and the group of operationbuttons 120 on the casing top surface 122.

Provided on a front side of the controller 102 are a right-handoperation part 150 and a left-hand operation part 160. The right-handoperation part 150 includes an R1 button and an R2 button, whereas theleft-hand operation part 160 includes an L1 button 161 and an L2 button162.

The user can shift a pointer displayed on the screen in vertical andhorizontal directions by operating the directional key group 116. Forexample, when selecting one of a plurality of markers displayed within apanoramic image, the user can shift the pointer between the plurality ofmarkers on the screen by operating the directional key group 116. Theuser can select a desired marker by pressing the circle button 124 whenthe pointer has come upon the marker.

Different functions may be assigned to the respective buttons ofoperation buttons 120 by a panoramic image display application program.For example, the function to specify the display of a menu is assignedto the triangle button 130, the function to specify the cancel of aselected item is assigned to the cross button 126, the function tospecify the determination of a selected item is assigned to the circlebutton, and the function to specify the display/non-display of table ofcontents or the like is assigned to the square button 128.

The analog sticks 118 have means to output analog values as they aretipped by the user. The controller 102 sends an analog output signalcorresponding to the direction and amount of tipping of the analog stick118 to the panoramic image display apparatus 100. For example, the usercan shift the viewpoint in a desired direction within a 3D panoramicimage shown on the display by tipping the analog stick 118 in thedesired direction.

The casing top surface 122 is further provided with an LED button 136, aselector button 140, and a start button 138. The LED button 136 is usedas the button for the display of the menu screen on the display, forinstance. The start button 138 is the button with which the userinstructs the start of a panoramic image display application, the startor pause of playback of a panoramic image, or the like. The selectorbutton 140 is the button with which the user instructs a selection froma menu display shown on the display or the like.

FIGS. 3A to 3D are illustrations with which to explain the mechanism andshooting directions of an omnidirectional image shooting system 230 usedto shoot panoramic images.

As shown in FIG. 3D, a camera 200 in the omnidirectional image shootingsystem 230 is secured onto a control disk 210. And a camera's pan anglecan be changed as the control disk 210 is rotated around a Z axis, acamera's tilt angle can be changed as the control disk 210 is rotatedaround an X axis, and a camera's roll angle can be changed as thecontrol disk 210 is rotated around a Y axis. The Z axis herein is thevertical axis (gravitational direction axis).

FIG. 3A is a top view of the camera 200 installed on the control disk210. The initial position (Y-axis direction) of the control disk is panangle 0°, and the pan angle can be changed within a range of −180° to+180° around the Z axis.

FIG. 3B is a front view of the camera 200 installed on the control disk210. The horizontal state of the control disk 210 is roll angle 0°, andthe roll angle can be changed within a range of −180° to +180° aroundthe Y axis.

FIG. 3C is a side view of the camera 200 installed on the control disk210. The horizontal state of the control disk 210 is tilt angle 0°, andthe tilt angle can be changed within a range of −90° to +90° around theX axis.

In order to endow a panoramic image shot by the omnidirectional imageshooting system 230 of FIG. 3D with information on the shootingorientations, it is necessary to record the orientations of the camera200 at the time of image taking. For that purpose, the omnidirectionalimage shooting system 230 is provided with an azimuth sensor formeasuring orientations and an acceleration sensor for measuring tiltangles. The ominidirectional image shooting system 230 is furtherprovided with a GPS sensor or the like for measuring the shootinglocation and time.

FIG. 4A is an illustration with which to explain azimuth angle θ of thecamera 200, and FIG. 4B is an illustration with which to explainelevation angle φ of the camera 200. FIG. 4A is a top view of the camera200, in which the camera 200 in an initial position of shooting faces adirection 220 which is azimuth angle θ displaced from true north toeast. This direction is equal to pan angle 0°. In other words, theazimuth angle of the reference direction 220 of the pan angle is θ. Whenshooting a panoramic image, the image of an object is takenpanoramically by changing the pan angle in a range of −180° to +180°with respect to the reference direction 220 of the azimuth angle θ.

FIG. 4B is a side view of the camera 200. The elevation angle φ is thedirection of tilt 0°, which is an angle where an upper direction isdefined to be positive in relation to the Y-axis direction, when thecamera 200 is rotated around the X axis. Normally, the elevation angle φis 0° since the image taking is done with the camera 200 set in ahorizontal position. To shoot a spherical panoramic image, however, itis necessary to take the images of the object by changing the elevationangle φ with the tilt of the camera.

FIGS. 5A to 5C are illustrations with which to explain a panoramic imageshot when the initial position of the camera 200 is in a direction ofthe azimuth angle θ.

As shown in the top view of FIG. 5A, the camera 200 in the initialposition faces the direction 220 of azimuth angle θ. And as shown in theside view of FIG. 5B, the elevation angle of the camera 200 is φ=0°.With the elevation angle kept at φ=0°, an omnidirectional panoramic viewis shot at the elevation angle φ=0° while the pan angle of the camera200 with respect to the reference direction 220 is varied within a rangeof −180° to +180°. FIG. 5C is a panoramic image 300 taken in theabove-described manner. At the center of the panoramic image 300, thepan angle is 0°. The left half of the panoramic image 300 is an imagesuch that it is taken by varying the pan angle within a range of 0° to−180°. Similarly, the right half of the panoramic image 300 is an imagesuch that it is taken by varying the pan angle within a range of 0° to180°.

The central position of the pan angle 0° is displaced from true north byazimuth angle θ. Thus, the positions of north (N), south (S), east (E),and west (W) are those indicated by dotted lines. As long as thepanoramic image 300 contains the azimuth angle θ of the central positionof pan angle 0° as the information on the shooting orientations, thepixel positions of north (N), south (S), east (E), and west (W) can beevaluated in consideration of a displacement of the azimuth angle θ.Alternatively, instead of the azimuth angle θ, the coordinate values ofpixel positions of north (N), south (S), east (E), and west (W) may beused as the information on the shooting orientations.

In order to obtain a spherical panoramic image, it is necessary to takeimages by varying the elevation angle of the camera 200. For example, ifthe angle of view of the camera 200 is 60°, a spherical panoramic imagecan be theoretically obtained as follows. That is, the camera 200 istilted vertically at ±60°, and the similar image taking is done byvarying the pan angle within a range of −180° to +180°.

FIGS. 6A to 6C are illustrations with which to explain a panoramic imageshot when a camera 200 is in a direction of elevation angle φ=60°. Asshown in the top view of FIG. 6A, the camera 200 in the initial positionfaces the direction 220 of azimuth angle θ. And as shown in the sideview of FIG. 6B, the elevation angle of the camera 200 is φ=0°. With theelevation angle kept at φ=60°, a panoramic view 302 as shown in FIG. 6Cis shot at the elevation angle φ=60° while the pan angle of the camera220 with respect to the reference direction 220 is varied within a rangeof −180° to +180°.

With the elevation angle kept at φ=−60°, a panoramic view 302 issimilarly shot at the elevation angle φ=−60° while the pan angle isvaried within a range of −180° to +180°. A spherical panoramic image isobtained by combining the panoramic images shot at the elevation anglesφ=0°, 60°, and −60°. However, in implementation, a method is oftenemployed where the vicinities of a boundary (bordering areas) are takenin an overlapped manner, in order to correct the mismatch caused by lensdistortions when images are stitched together in boundary portions atthe angle of view.

The spherical panoramic image obtained as described above is endowedwith information on azimuth angles and elevation angles. Therefore, itis possible to identify the azimuth and elevation angle of an arbitrarypixel of the panoramic image based on the information. Also, thepanoramic image is provided with the latitude and longitude informationmeasured by GPS as the positional information of the shooting location.The additional information to be attached to the panoramic image may berecorded, for example, in the format of image file called Exif(Exchangeable Image File Format). The place-name of the shootinglocation can be recorded in a part of the file name, whereas theshooting date and time, the latitude and longitude of the shootinglocation, the altitude, the azimuth angle, and the like can be recordedas data in the Exif format. The elevation angle, which is not defined inthe Exif format, is recorded as extended data.

FIG. 7A and FIG. 7B are illustrations with which to explain a method ofcreating a panoramic image by stitching a plurality of images together.

In the example of FIG. 7A, seven images 341 to 347 shot by tilting (orpanning) the camera 200 are mapped into a cylinder and then stitchedtogether to prepare a cylindrical image 340. When the images arestitched together, the bordering areas of the images are overlapped witheach other.

As shown in FIG. 7B, a plurality of cylindrical images like one shown inFIG. 7A are obtained in the panning (or tilting) direction by theshooting with the panning (or tilting) of the camera 200. Anomnidirectional panoramic image 360 is finally obtained by synthesizingthese cylindrical images 340 a to 340 f with the bordering areas of theimages overlapped.

FIG. 8 is a flowchart showing a procedure for generating a panoramicimage by the panoramic image display apparatus 100. With reference toFIGS. 9A to 9E, FIG. 10 and FIGS. 11A and 11B, each step in theprocedure for generating a panoramic image of FIG. 8 will be explained.In the flowchart shown in FIG. 8, the procedure of each structuralcomponent is shown using S (the capital letter of “Step”), which means astep, and numbers combined.

The panoramic image acquiring unit 10 acquires information on apanoramic image 400 shot and information on shooting orientationsappended to the panoramic image 400 from the panoramic image/additionaldata storage 24 (S10).

FIG. 9A shows an example of a panoramic image 400 shot. A building iscaptured in the center of the panoramic image, and the ground surface iscaptured in a lower part of the panoramic image. Sky 410 is captured inan upper part of the panoramic image, and clouds as well as the sun 430are captured in the sky 410.

The mask region extraction unit 11 extracts the sky region of thepanoramic image 400 as the region to be masked (S12). In order toextract the sky region from the panoramic image, the mask regionextraction unit 11 uses a technique called segmentation and therebysegmentalizes or partitions the panoramic image into meaningfulsegments. The segmentation is a technique often used in the field ofimage recognition. In this technique of segmentation, regionscorresponding respectively to objects within a given image are detectedand the image is segmented. By the segmentation, the image is segmentedinto the regions for the respective objects, such as sky, building,ground, and persons.

Generally, the segmentation uses a clustering method. Since the pixelsassociated with the objects captured in the image share similarcharacteristics in color, brightness and the like, the image can besegmented into the regions corresponding respectively to the objects byclustering those pixels. A supervised clustering that gives correctanswers as training data may be used. In the present embodiment, thetraining data is given when the user specifies several pixels belongingto the sky region in the panoramic image.

FIG. 9B shows a panoramic image 402 that is partitioned into segmentedimages by the segmentation. One can recognize that the panoramic imageis segmentalized into the sky region, building region, ground region,and so forth by the segmentation.

As for the sky region 410, the sky region 410 is partitioned mainly intofive regions denoted by the reference numerals 411 to 415, as shown inFIG. 9B, by the segmentation. Since the sun 430 is captured in theoriginal panoramic image 400, the sky region 410 is detected as the fiveregions whose color vary in stages from blue to white in the order ofthe regions 411 to 415 and the sky region is not detected as a singleregion. Also, in the region 411, the region corresponding to the cloudsis extracted as another segment and therefore the segment for the sky isnot perfectly detected.

Nevertheless, a method is implemented, as follows, even though thesegmentation result is incomplete as mentioned above. That is, if thecolor of the sky is specified and if the absolute value of differencebetween the color of pixel for each region and the specified color ofthe sky is less than a predetermined threshold value, the region will bedetermined to be the sky region. Thereby, for example, three regions411, 412, and 413 out of the five regions may be detected as a singlesky region altogether. Also, the region corresponding to the cloudswithin the region 411 may be deleted manually.

The segmentation is generally a time-consuming process. Thus thesegmentation technique may be suitable for its use if the processing ofpanoramic images stored in the recording device can be executed usingthe idle time but it may not be very suitable if the processing ofpanoramic images shot has to be processed in real time. In the light ofthis, the mask region extraction unit 11 may use a filtering process,using a so-called bilateral filter, as alternative technique in place ofthe segmentation.

FIG. 9C shows a panoramic image 404 that has been filtered. The pixelsof the panoramic image 400 of FIG. 9A are planarized and the panoramicimage 400 of FIG. 9A is also subjected to the bilateral filter throughwhich edges are enhanced or preserved. As a result, as shown in FIG. 9C,the gradation of sky and the clouds disappear and therefore the skyregion is detected as a single region 421. The region in which the sun430 is captured is extracted as separated regions 422 and 423, havinggradations therein, for instance.

If the absolute value of difference between the color of each pixel ofthe filtered panoramic image 404 and the color of the sky is less than apredetermined threshold value, the mask region extraction unit 11 willdetermine that the pixel belongs to the sky and thereby extract the skyregion from the panoramic image 404. The color of the sky may bespecified by the user or may be automatically set according to theweather or season at the time of image taking.

If, however, in this determination method, the color of pixels is closeto the color of the sky, said pixels of an object other than the sky maybe mistakenly selected as those belonging to the sky region. In order toavoid this problem, the mask region extraction unit 11 detects thehorizon position in the panoramic image using the information on theelevation angle of the camera at the time of shooting, so that eventhough there is any pixel close to the color of the sky below thehorizon, such a pixel is not detected as that belonging to the skyregion. As a result, if, for example, the sea, pond, swimming pool orlike object is captured in the panoramic image, the error of mistakenlydetermining the object to be the sky can be avoided.

Also, it is often the case that the sky is opened up above a persontaking the image of a panoramic view, and the sky region is usually acontinuously extending region having a certain large amount of space.Thus, the number of areas mistakenly determined to be the sky can beminimized with the help of knowledge that the sky region is generallycontinuous.

In the example of FIG. 9C, the mask region extraction unit 11 extractsregions 421, 422, and 423 including the region 422 and 423 thatcorrespond to the sun 430, as the sky region.

FIG. 9D shows a mask 440 by which the sky region in the panoramic imageis specified. The mask region extraction unit 11 sets a sky regionextracted from the filtered panoramic image 404 as the mask 440 andsupplies the thus set sky region to the mask processing unit 12.

Referring back to the flowchart of FIG. 8, the mask processing unit 12performs the mask processing on the panoramic image 400 using the mask440 set by the mask region extraction unit 11 so as to clip the skyregion (S14).

FIG. 9E shows a panoramic image 408 obtained after the sky region hasbeen mask-processed. FIG. 9E is the image obtained when the sky region410 has been clipped from the original panoramic image 410.

Next, the positioning unit 20 aligns the shooting direction of thepanoramic image mapped onto the 3D panoramic space and the direction ofthe spherical image mapped onto the same 3D panoramic space with eachother (S16). The object insertion unit 22 places an object to beinserted outside or inside the 3D panoramic space as necessary (S18).

FIG. 10 is a diagram for explaining a relation between a sphere and a 3Dpanoramic image 408 into which the mask-processed panoramic image 408 ismapped.

A panoramic image 510, whose sky region 512 has been made transparent bythe mask processing, is mapped into a panoramic sphere 500 that is anexemplary 3D panoramic space. The center of the panoramic sphere 500 isthe shooting location of the panoramic image, namely the viewpointposition. A sphere (celestial sphere) 540 is provided outside thepanoramic sphere 500 with the viewpoint position of the panoramic sphere500 as the center. A spherical image including clouds 532 and 534, thesun and the like are set on the surface of the sphere 540. The sphericalimage may be selected from the images stored in the spherical image datastorage 26, and a spherical image of starry sky where stars and the moonare rendered may be used.

The panoramic image 510 mapped into the panoramic sphere 500 and thespherical image set on the surface of the sphere 540 are adjusted byrotating the panoramic sphere, into which the panoramic image is mapped,in such a manner that the orientations of their North Stars agree witheach other. If the direction may be ignored, the processing ofpositioning the panoramic image 510 and the spherical image will not berequired. If, however, no processing of positioning the panoramic image510 and the spherical image is done, the sun, the moon and starsrendered on the celestial space will appear in a direction differentfrom the shooting direction of the panoramic image and therefore acomposite image may look unnatural as a whole.

An airship 520 and an airplane 530 are exemplary objects inserted intothe panoramic image. The airplane 530 is placed between the sphere 540and the panoramic sphere 500. In other words, the airplane 530 is placedoutside the panoramic sphere 500. The airship 520 is placed inside thepanoramic sphere 500. If the user wishes to insert an object at adistant view, the object will be placed outside the panoramic sphere500. If he/she wishes to insert the object at a new view, the objectwill be placed inside the panoramic sphere 500. The airship 520 and theairplane 530 move within the sphere 540 according to their motionmodels.

Referring back to FIG. 8, the mapping processing unit 14 maps themask-processed panoramic image 510 and the spherical image rendered inthe sphere 540 onto the panoramic sphere 500. Also, the mappingprocessing unit 14 maps the objects, such as the airship 520 and theairplane 530, onto the panoramic sphere 500 (S20).

The sky region of the panoramic image 512 is clipped and is now atransparent region as viewed from the center of the panoramic sphere500, namely the viewpoint position. Thus, a spherical image rendered onthe surface of the sphere 540 is seen in the sky region 512. Theairplane 530 located outside the panoramic sphere 500 is seen at a farpoint beyond the sky region 512 but cannot be seen when the airplane 530is blocked by the building captured in the panoramic image. At the sametime, the airship 520 located inside the panoramic sphere 500 is seen ata short-distant point and may be seen in front of the building in somecases.

In order to reproduce an image as viewed from the center of thepanoramic sphere 500, the mapping processing unit 14 maps images,starting from an image located outermost from the viewpoint, into thepanoramic sphere 500 and renders them in a superimposed manner. First, aspherical image rendered in the sphere 540 is mapped into the panoramicsphere 500 and, on top of it, the airplane 530 placed outside thepanoramic sphere 500 is mapped into the panoramic sphere 500. Then themask-processed panoramic image 510 is mapped into the panoramic sphere500. Finally the airship 520 placed inside the panoramic sphere 500 ismapped into the panoramic sphere 500. As a result, the spherical imageis pasted to the transparent sky region 512 of the panoramic image 510so that the spherical image can be peeked through the sky region 512.Also, the airplane 530 outside the panoramic sphere 500 is pasted to adistant view and the airship 520 inside the panoramic sphere 500 ispasted to a near view.

FIG. 11A shows an example where the spherical image has been pasted tothe sky region of the mask-processed panoramic image 408 of FIG. 9E. Aspherical image including the clouds 532 and 534 is pasted to the skyregion of the panoramic image 450 of the FIG. 11A. Also, the airplane530 is inserted to a distant view, and the airship 520 is inserted to anew view in front of the building.

FIG. 11B shows, as another exemplary spherical image, an example where astarry sky image has been pasted to the mask-processed panoramic imageof FIG. 9E. A spherical image including the moon and stars is pasted tothe sky region of the panoramic image 408. The phases of the moon andthe constellation of stars in the sky are adjusted according to the dataand time when the panoramic image is displayed, so that a realisticnight sky can be shown in the sky region of the panoramic image.

In the present embodiment, the sphere 540 is provided outside thepanoramic sphere 500 into which the panoramic image is mapped, so thatthe spherical image rendered in the sphere 540 can be changedindependently of the panoramic image and then the spherical image can becombined with the panoramic image.

For example, the user may adjust the motion of constellation of starsrendered in the celestial sphere by operating the R1/R2 buttons 151 and152 of the controller 102. For example, pressing the R1 button 151 willadvance date and time, and pressing the R2 button 152 will return dateand time, so that the constellation of starts in different reasons canbe displayed in the sky region.

Referring back to the flowchart of FIG. 8, the 3D image generator 16generates a 3D panoramic image when a 3D panoramic sphere having themask-processed panoramic image 510, an image of the sphere 540 andobjects such as the airship 520 and the airplane 30 mapped thereon isviewed in a specified line of sight, and the display control unit 18displays the thus generated 3D panoramic image (S22).

As described above, by employing the panoramic image display apparatusaccording to the present embodiment, the sky region of a panoramic imagecan be clipped and replaced with another spherical image. For example,even though the panoramic image is shot on a rainy day, the sky regioncan be clipped and then replaced with a fine blue sky or a sky with thesun rising up can be changed to a sky with the sun setting down, therebybringing diversity to the panoramic image after the actual image taking.Also, a virtual object such as an airplane and a satellite can beinserted to the panoramic image, thus excelling in quality ofentertainment. The present embodiment may be used in the technical fieldof so-called augmented reality (AR) in which virtual world informationand real world information are merged with each other. That is, apanoramic image, such as a starry sky, which is created virtually basedon a real-world panoramic image is superimposed onto the real-worldpanoramic image, so that a virtual real world in which the realenvironment is augmented by the virtual information can be created.

The present invention has been described based upon illustrativeembodiments. These embodiments are intended to be illustrative only andit will be obvious to those skilled in the art that variousmodifications to the combination of constituting elements and processescould be developed and that such modifications are also within the scopeof the present invention.

In the foregoing description, the mask-processed panoramic image and thespherical image are mapped into a 3D panoramic image, such as thesurface of a sphere, and then a 3D image, when the 3D panoramic space isviewed in a specified line of sight, is displayed on the screen.However, an image obtained when a virtual image is superimposed onto themask region may be simply displayed two-dimensionally. In this case, themapping processing unit 14 and the 3D image generator 16 are no longernecessary, thus simplifying the panoramic image display apparatus 100.Not only the virtual image may be merged into or combined with thepanoramic image off-line but also an augmented-reality image in whichthe mask region of the panoramic image has been replaced with thevirtual image on-line may be displayed on a display device such as atransparent display wearable by the user.

In the above-described embodiments, the sky region of a panoramic imageshot outdoors is extracted as the region to be masked. If, for example,the panoramic image is one for which the sports activity like baseballis shot in a stadium with a dome-shaped roof, the region of thedome-shaped roof may be extracted as the region to be masked. Clippingthe region of the dome-shape region and then pasting a spherical imageto the region can make the image seem as if the baseball were beingplayed outdoors. In clipping the region of the dome-shaped region, aregion shot at a predetermined elevation angle of the camera or a regionat an altitude of more than a predetermined altitude is preferablyextracted as the region to be masked, for instance. Also, a region whosecolor barely changes or a region having a predetermined area may beextracted as the region to be masked. For example, windowpanes of abuilding may be masked, then the regions corresponding to the windowsmay be clipped, and another image may be pasted thereonto. A buildingwhere no people are may be virtually seemed as if there were peopleinside, or the scene outside the building that can be seen from thewindow may be changed. Also, the sea, pond or desert may be extracted asthe region to be masked. Tokyo Bay may be replaced by South PacificOcean or a desert may be replaced by the image of green land.

Panoramic images herein are not limited to ones shot by theomnidirectional image shooting system as shown in FIG. 3, but they maybe ones shot through a fish-eye lens or ones merged or brought togetherfrom a plurality of images shot with a normal digital camera indifferent shooting directions.

In the above-described embodiments, when the mask-processed panoramicimage and the spherical image are to be displayed such that they aresuperimposed on each other, the images are rendered such that theimages, starting from an image located outermost from the viewpoint, aremapped into the 3D panoramic sphere. That is, the spherical image ismapped to the 3D panoramic sphere and then the panoramic image, wherethe mask region has been clipped and therefore the clipped region istransparent, is mapped into the 3D panoramic sphere in a superimposedmanner. Another mapping method may be implemented instead. That is, aregion corresponding to the mask may be clipped from the spherical imageand then a spherical image corresponding to the mask may be pasted ontothe mask region of the panoramic image mapped into the 3D panoramicsphere.

In general, image discontinuity occurs on a boundary portion of the maskregion because of the synthesis of the panoramic image and the sphericalimage, thus giving an unnatural impression. In the light of this, animage of the mask region may be passed through a Gaussian filter so asto be smoothed out. And the panoramic image and the spherical image maybe alpha-blended near the boundary of the mask region, so that theboundary may be made blurry.

Of the functional components of the panoramic image display apparatus100, the components related to a function for mainly performing themasking processing of a panoramic image and superimposing a sphericalimage and/or virtual image onto the mask region of the panoramic imagemay be implemented to a server. And the components related to a functionfor mainly viewing the panoramic image onto which the spherical imageand/or virtual image have/has been superimposed may be implemented to aclient. Thereby, the panoramic image display apparatus 100 can berealized as a server-client system via a network. The server may carryout the mask processing after the mask region of panoramic image hasbeen extracted and then superimpose the spherical image and/or virtualimage onto the mask region. And an interface with which to view thepanoramic view the panoramic image, onto which the spherical imageand/or virtual image have/has been superimposed, may be provided to theclient. There may be provided an interface with which the user specifiesa specific region of the panoramic image on a display screen as a regionto be masked. Also, the client may receive the data of the panoramicimage and the spherical image from the server, adjust the positioning ofthe spherical image and the panoramic image, as appropriate, andsuperimpose the spherical image onto the panoramic image. This allowsthe user to freely rotate the spherical image and allows the thusrotated panoramic image to be superimposed onto the panoramic image.

The “panoramic image” as used in this patent specification is notlimited to a “panorama” image in the narrow sense, namely a horizontallyor vertically long image or a panoramic view of 360 degrees, but simplyan image that covers a wide range of area. Also, explained in theembodiments is an example where the moving images are pasted on thepanoramic image, which is an object image. Note that the object imagedoes not have to be a so-called panoramic image and that the presentinvention is applicable to a normal image of arbitrary size serving asthe object image. Or the object image may be an image where a pluralityof images having different resolutions are hierarchized. Such ahierarchized image may be constructed such that when a partial region ofimage is enlarged, the enlarged region is replaced by an image of ahigher resolution. Also, “spherical image (celestial image)” is notnecessarily limited to an image assuming an object image is of a“sphere” or a “spherical surface” but simply an image including the skyand the like.

The invention claimed is:
 1. An image display apparatus comprising: astorage configured to store an object image, associated with informationon a shooting direction, a first virtual spherical image having a firstthree dimensional (3D) inner surface, associated with first directionalinformation, and a second virtual spherical image having a second 3Dinner surface, associated with second directional information, where thefirst virtual spherical image and the second virtual spherical image arein the form of spheres; a mask region extraction unit configured toextract a region that is to be masked in the object image; a maskprocessing unit configured to generate the object image where the regionto be masked in the object image is masked; a positioning unitconfigured to adjust the first directional information of the firstvirtual spherical image to be the shooting direction of the objectimage; a mapping processing unit configured to: (i) position the firstvirtual spherical image centered within the second virtual sphericalimage about a central point such that the first directional informationand the second directional information align, (ii) map themask-processed object image onto the first 3D inner surface of the firstspherical virtual image such that the shooting direction and the firstdirectional information are aligned, (iii) map at least one firstvirtual object onto the first 3D inner surface of the first virtualspherical image when a distance from the central point to the at leastone first virtual object is within the first virtual spherical image,and (iv) map at least one second virtual object onto the second 3D innersurface of the second virtual spherical image when a distance from thecentral point to the at least one second virtual object is outside thefirst virtual spherical image such that the at least one second virtualimage appears in the mask area from the viewer's point of view in theshooting direction in a three-dimensional (3D) object space; athree-dimensional (3D) image generator configured to generate athree-dimensional (3D) object image, when the 3D object image mapped bythe mapping processing unit is viewed in a specified line of sight insuch a manner so as to regard a shooting location of the object image asa viewpoint position; and a display control unit configured to displaythe 3D object image on a screen.
 2. An image display apparatus accordingto claim 1, wherein the mask region extraction unit extracts a skyregion in the object image, as the region to be masked, and wherein themask processing unit generates the object image where the sky region inthe object image is masked.
 3. An image display apparatus according toclaim 2, wherein the mask region extraction unit determines a horizonposition in the object image based on information on an elevation angleof a camera at the time of shooting, and the mask region extraction unitextracts the sky region in a region above the horizon position.
 4. Animage display apparatus according to claim 1, wherein the mappingprocessing unit generates an image onto the 3D object space, in whichpart of the spherical image is pasted onto the region to be masked inthe object image, by mapping the mask-processed object image onto the 3Dobject space in a manner such that the mask-processed object image issuperimposed onto the 3D object space where the spherical image ismapped.
 5. An image display apparatus according to claim 1, furthercomprising an object insertion unit configured to place the at least onevirtual object onto the one of the first and second 3D inner surfacesbased on the distance from the central point, wherein the mappingprocessing unit maps the at least one object onto the 3D object space.6. An image display apparatus according to claim 5, wherein, when the atleast one virtual object is placed onto the second 3D inner surface,which is outside the 3D object space, the mapping processing unitgenerates an image onto the 3D object space, in which the object issynthesized with the object image, by mapping the mask-processed objectimage onto the 3D object space in a manner such that the mask-processedobject image is superimposed onto the 3D object space where the objectis mapped.
 7. An image display apparatus according to claim 5, wherein,when the at least one virtual object is placed onto the first 3D innersurface, which is inside the 3D object space, the mapping processingunit generates an image onto the 3D object space, in which the object issynthesized with the object image, by mapping the object onto the 3Dobject space in a manner such that the object is superimposed onto the3D object space where the mask-processed object image is mapped.
 8. Animage display apparatus according to claim 1, wherein the object imageis a panoramic image and the 3D object space is a 3D panoramic space. 9.An image display apparatus comprising: a storage configured to store anobject image, associated with information on a shooting direction, afirst virtual image having a first 3D inner surface, associated withfirst directional information, and a second virtual image having asecond 3D inner surface, associated with second directional information,where the first virtual image and the second virtual image are in theform of spheres; a mask region extraction unit configured to extract aregion that is to be masked in the object image; a mask processing unitconfigured to generate the object image where the region to be masked inthe object image is masked; a positioning unit configured to adjust thefirst directional information of the first virtual image to the shootingdirection of the object image; and a display control unit configured to:(i) position the first virtual image centered within the second virtualimage about a central point such that the first directional informationand the second directional information align, (ii) map themask-processed object image onto the first 3D inner surface of the firstvirtual image such that the shooting direction and the first directionalinformation are aligned, (iii) map at least one first virtual objectonto the first 3D inner surface of the first virtual spherical imagewhen a distance from the central point to the at least one first virtualobject is within the first virtual spherical image, (iv) map at leastone second virtual object onto the second 3D inner surface of the secondvirtual spherical image when a distance from the central point to the atleast one second virtual object is outside the first virtual sphericalimage such that the at least one second virtual image appears in themask area from the viewer's point of view in the shooting direction in athree-dimensional (3D) object space, and (v) display on a screen anaugmented-reality image where the virtual image is superimposed onto themask-processed object image.
 10. An image display apparatus according toclaim 9, wherein the object image is a panoramic image.
 11. An imagegeneration method comprising: reading out, by a processor, an objectimage, a first virtual spherical image, and a second virtual sphericalimage to be displayed, from a storage device for storing the objectimage, associated with information on a shooting direction, the firstvirtual spherical image having a first 3D inner surface, associated withfirst directional information, and the second virtual spherical imagehaving a second 3D inner surface, associated with second directionalinformation, where the first virtual spherical image and the secondvirtual spherical image are in the form of spheres such that each of thefirst and second virtual spherical images are round and characterized inthat all points of a surface thereof are equidistant from a centerthereof; specifying, by a processor, a region that is to be masked inthe object image; adjusting, by a processor, the first directionalinformation of the first virtual spherical image to the shootingdirection of the object image; positioning the first virtual sphericalimage centered within the second virtual spherical image about a centralpoint such that the first directional information and the seconddirectional information align; mapping the mask-processed object imageonto the first 3D inner surface of the first spherical virtual imagesuch that the shooting direction and the first directional informationare aligned; mapping at least one first virtual object onto the first 3Dinner surface of the first virtual spherical image when a distance fromthe central point to the at least one first virtual object is within thefirst virtual spherical image; mapping at least one second virtualobject onto the second 3D inner surface of the second virtual sphericalimage when a distance from the central point to the at least one secondvirtual object is outside the first virtual spherical image such thatthe at least one second virtual image appears in the mask area from theviewer's point of view in the shooting direction in a three-dimensional(3D) object space; and rendering, by a processor, the first virtualspherical image superimposed onto the region to be masked in the objectimage.
 12. A non-transitory computer-readable medium encoded with aprogram, the program comprising: a reading module operative to read outan object image, a first virtual spherical image, and a second virtualspherical image to be displayed, from a storage device for storing theobject image, associated with information on a shooting direction, thefirst virtual spherical image having a first 3D inner surface,associated with first directional information, and the second virtualspherical image having a second 3D inner surface, associated with seconddirectional information, where the first virtual spherical image and thesecond virtual spherical image are in the form of spheres such that eachof the first and second virtual spherical images are round andcharacterized in that all points of a surface thereof are equidistantfrom a center thereof; a mask region extraction module operative toextract a region that is to be masked in the object image; a maskprocessing module operative to generate the object image where theregion to be masked in the object image is masked; a positioning moduleoperative to adjust the first directional information of the firstvirtual spherical image to the shooting direction of the object image; apositioning module operative to position the first virtual sphericalimage centered within the second virtual spherical image about a centralpoint such that the first directional information and the seconddirectional information align; a mapping processing module operative to:(i) map the mask-processed object image onto the first 3D inner surfaceof the first spherical virtual image such that the shooting directionand the first directional information are aligned, and (ii) map at leastone first virtual object onto the first 3D inner surface of the firstvirtual spherical image when a distance from the central point to the atleast one first virtual object is within the first virtual sphericalimage, and (iii) map at least one second virtual object onto the second3D inner surface of the second virtual spherical image when a distancefrom the central point to the at least one second virtual object isoutside the first virtual spherical image such that the at least onesecond virtual image appears in the mask area from the viewer's point ofview in the shooting direction in a three-dimensional (3D) object space;and a 3D image generation module operative to generate athree-dimensional (3D) object image, when the three-dimensional objectimage mapped by the mapping processing module is viewed in a specifiedline of sight in such a manner so as to regard a shooting location ofthe object image as a viewpoint position.